Skip to content

csdb: compaction of pruned storage can no longer terminate the node - #25

Merged
EnzinoBB merged 3 commits into
masterfrom
fix/prune-compact-memory
Oct 12, 2026
Merged

EnzinoBB merged 3 commits into
masterfrom
fix/prune-compact-memory

Conversation

@EnzinoBB

Copy link
Copy Markdown
Collaborator

Problem

A pruned node (#22) was started on mainnet from block 0, as an observer in a container. Its first pruning cycle terminated it: at the 200,000 checkpoint it removed the blocks before 99,000 (fine), then compacted the ~600 MB blockchain.db in a single Db::compact call:

BDB3017 unable to allocate space from the buffer cache
terminate called after throwing an instance of 'DbException'
  what():  Db::compact: Cannot allocate memory

The test network never showed this, because its databases are a few MB.

Fix

  • DatabaseBerkeleyDB::compact: works through each database 64 pages per call, resuming from the key where the previous call stopped. Any exception becomes an error result; it can no longer terminate the node.
  • Storage::compact: a failure is logged as a warning. It is not a storage fault: compaction only returns space to the file system.

What mainnet shows with the fix

The node restarted from its pruned storage ("oldest block stored is 99'000") and kept syncing.

  • 300,000 checkpoint: blocks before 199,000 were removed. Compaction was again refused for lack of cache, with no consequence. While a node syncs, BerkeleyDB's default buffer cache (~64 pages of 4 KB) is full of freshly written pages.
  • Space stays bounded without compaction: pruning frees pages and BerkeleyDB reuses them for new blocks. blockchain.db grew from 207 MB to 232 MB while 100,000 blocks were added and 100,000 removed.

Not changed here: making compaction succeed during sync would need a larger BerkeleyDB cache (set_cachesize), which changes every node's memory use. That needs measuring first.

🤖 Generated with Claude Code

EnzinoBB and others added 3 commits October 12, 2026 07:14
On mainnet the first pruning cycle of a node syncing from block 0 (blocks before 99'000 removed
at the 200'000 checkpoint) terminated the node: compacting the 600 MB blockchain.db in one
Db::compact call needed more of the environment's buffer cache than it holds, and BerkeleyDB
threw (BDB3017 unable to allocate space from the buffer cache; terminate called after throwing
an instance of 'DbException', what(): Db::compact: Cannot allocate memory).

DatabaseBerkeleyDB::compact now works through each database 64 pages per call, resuming from the
key the previous call stopped at, and turns any exception into an error result: compaction only
returns space to the file system (BerkeleyDB reuses freed pages anyway), so pruneStep logs the
failure and the node carries on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BerkeleyDB's default buffer cache (about 64 pages of 4 KB) is usually full of freshly written
pages while a node syncs, and Db::compact then cannot allocate pages even in small steps
(seen on mainnet at the 300'000 checkpoint). Nothing is wrong with the storage: compaction only
returns space, and the pages freed by pruning are reused for new blocks (blockchain.db grew from
207 to 232 MB while 100'000 blocks were added and 100'000 removed). Log it as a warning instead
of setting the storage error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On mainnet a pruned node syncing fast terminated in Db::del: BerkeleyDB's default buffer cache
was exhausted (BDB3017) and the exception escaped DatabaseBerkeleyDB::remove. remove now turns
any exception into an error result, and pruneStep keeps a block it could not remove and retries
it on the next step instead of counting it as pruned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit 401d3dc into master Oct 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant